FESTA. Handbook version 2

In Japan and in the United States Field Operational Tests (FOTs) have been introduced as an evaluation method for driver support systems and other functions several years ago with the aim of proving that such systems can deliver real‐world benefits. In Europe too, FOTs have been conducted at a national or regional level, particularly on speed support systems and lane departure warning systems. These FOTs have proven to be highly valuable. Recently FOTs have been identified as an important means of verifying the real‐world impacts of new systems at a European level and in particular to verify that European R&D has the potential to deliver identifiable benefits. This Handbook is the result of a joint effort of several research institutes, OEMs and other stakeholders from across Europe to prepare a common methodology for European FOTs. It is also highly relevant, and it is hoped useful, for FOTs conducted at a regional or national level within Europe as well as outside Europe

FESTA. D3 Common vision regarding vehicle systems FOTs

Individuating the most relevant functions and connected hypothesis to successfully address the above-mentioned research questions is one of the major challenges in a Field Operational Test (FOT). In this deliverable, the process of individuating the vehicle functions to be tested in an FOT and the relevant connected hypotheses will be elucidated. Specifically, the reader will be guided in the process of: 1) selecting the vehicle functions to be tested, 2) defining the connected use cases to test these vehicle functions, 3) identifying the research questions related to these use cases, 4) formulating the hypothesis associated to these research questions, and 5) linking these hypothesis to the correspondent performance indicators. Each of these steps needs to be undertaken sequentially. Everyone who is interested to undertake an FOT will find templates to define and describe the necessary details for all of the above mentioned steps in the Annex of this deliverable

FESTA. D4 Common vision regarding cooperative systems FOTs

The objective of an FOT is to evaluate in-vehicle functions based on Information Communication Technology (ICT) in order to address specific research questions. These research questions can be related to safety, environment, mobility, traffic efficiency, usage, and acceptance. By addressing the research questions, FOTs promise to furnish the major stakeholders (customers, public authorities, OEMs, suppliers, and the scientific community) with valuable information able to improve their policy-making and market strategies. Individuating the most relevant functions and connected hypothesis to successfully address the above-mentioned research questions is one of the major challenges in an FOT. In this deliverable, the process of individuating the vehicle functions to be tested in an FOT and the relevant connected hypotheses will be elucidated. Specifically, the reader will be guided in the process of 1) selecting the vehicle functions to be tested, 2) defining the connected use cases to test these vehicle functions, 3) identifying the research questions related to these use cases, 4) formulating the hypothesis associated to these research questions, and 5) linking these hypothesis to the correspondent performance indicators

FESTA. D2.6: Socio-economic impact assessment for driver assistance systems

This deliverable provides further advice on the methodology for socio-economic assessment of ICT based systems, within the Field Operational Tests (FOTs) being part-funded by the European Commission. It can be treated as a supplement to Chapter 9 of the FESTA Handbook. Use of a consistent methodology in the FOTs will maximise the comparability of the results across regions, ICT systems and individual FOTs. The goal of this deliverable is therefore to provide concise advice on how to carry out a socio-economic impact assessment, addressing issues likely to arise and giving references to more detailed guidance elsewhere, plus examples of good practice in existing (web) documents. Topics covered by this document include: • the assessment framework; • the approach to specific stakeholders as part of the wider picture; • scope of the assessment - which impacts should be included; • analysis methods for specific impacts; • financial analysis; • data needs. Data will be one of the keys to success for the FOTs. This deliverable provides advice on which data will need to be collected during the FOT itself, i.e. 'FOT-specific data' (with implications for the design of FOTs), as well as generic data needed to carry out a socio-economic assessment. The latter include values for accident and casualty reduction. This advice will be useful for: clients commissioning FOTs; consortia drawing up proposals for FOTs; and organisations carrying-out FOTs. It is assumed that a specialist in the area of socioeconomic impact assessment will carry out the analysis - so a full "tutorial" on socio-economic assessment is neither feasible nor necessary in this deliverable. We will refer to this specialist as the “analyst”. The advice was prepared by a group of European experts in the area of stand-alone and cooperative vehicle ICT systems and socio-economic impact assessment. Their experience was enriched by a literature review of over twenty state-of-the-art sources, covering methodologies and applications in studies and FOTs from Europe, Australia and the United States. Taking into account the findings of those studies and the experience of the team, an assessment methodology was specified: cost-benefit analysis (CBA) was chosen as the overall framework for assessment

FESTA. D6.3. FOT requirements, legal aspects planning and development

The FESTA Support Action Questionnaire to collect stakeholders' needs and indications on Field Operational Tests was a part of FESTA Task T6.1 – Analysis of the different stakeholders and their needs, relevant aspects and requirements on different level of FOT deployment. The questionnaire question sets were designed in cooperation between VTT and CRF during November-December 2007. The survey was performed by VTT through a web survey tool “SurveyMonkey”, which serves several purposes: e.g. designing surveys, collecting responses, sending invitations and reminders and downloading or analysing the results. The web survey was opened on 13th December 2007, right after the FESTA Stakeholder Workshop in Brussels. The opening of the survey was announced to the target group members by an email invitation which included an individual hyperlink to the survey. The recipient list for the invitation was mostly based on the Stakeholder Workshop attendance list that was supplemented by CRF and VTT with the other known FOT stakeholders. Two reminder email rounds were sent out to those who had received an invitation to the survey but had not responded – the first one on 21st December 2007 and the final one on 10th January 2008. The survey was closed on 24th January 2008. The results reported are synthesised from the answers given by the respondents on the questions, which are given in Appendix 1. Next to the presentation of the results, some conclusions are drawn

FESTA. D2.4 Data analysis and modelling

The chapter of the handbook and the deliverable on data analysis will provide guidance and general principles for - pre-testing to check the usability of the system and the feasibility of the evaluation process, - controlling the consistency of the chain and the precision with different sampling schemes, - modelling the impact for each indicators and for an integrated evaluation including a systemic and multidisciplinary interpretation of the effects, - integrating and controlling the quality of space-time data from various sources (numerical, video, questionnaires), - selecting the appropriate statistical techniques for data processing, PI estimation and hypothesis testing in accordance to the list of indicators and experimental design, - scaling up from experimental data and identified models to population and network level. Experimentalists stress the role and importance of a preliminary field test in FOT. Three main objectives have been defined to make a preliminary diagnosis of usability of the systems and to check the relevance and feasibility of the evaluation process. These preliminary tests are very important for the practical deployment of the FOT as well as for the overall scientific evaluation process. Recommendations about the monitoring of local and global consistency of the chain of operations from the database extraction to the hypothesis testing are given, especially to ensure the validation of the calculation of the Performance indicators. Integration of the outputs of the different analysis and hypothesis testing requires a kind of meta-model and the competences of a multidisciplinary evaluation team, specially for interpretation of the system impact and secondary effects (behavioural adaptation, learning process, long-term retroaction, …). In cooperation with WP2.2, methods for data quality control have been defined. Four types of checks have been defined to complement the information of the data base in order to prepare the data for the analysis. Statistical methods have been described for three steps of the chain: data processing, PI calculation and hypothesis testing. They belong either to exploratory data analysis or to inferential analysis. Special attention has been given to the precision of the estimates of the effects or impacts of the system on the Performance indicators by stressing the importance of controlled randomisation and application of mixed regression models. Scaling-up relies upon the potential to extrapolate from the PIs to estimates of the impact at an aggregated level. Three approaches have been defined to carry out the scaling up process from direct estimations to simulation models with the related assumptions. Models and methodologies for scaling up results on traffic flow, environmental effects (e.g. PM10, CO2, Noise emissions in db) and traffic safety have been collected

FESTA. D2.3 Primer on experimental procedures

This deliverable provides guidance on the overall experimental design of FOTs in order to ensure experimental rigour and scientific quality. The first section “Participants” provides advice on participant selection, including demographics, driving experience, personality and attitudes, along with consideration to sample size. The second section “Study design” provides guidance of the formulation of hypothesis, experimental design and possible confounds. The third section “Experimental environment” suggests how the road environment (road type, weather conditions etc.) plays a part in the design of an FOT and the subsequent data analysis

FESTA. D2.2. Data requirements for FOT Methodology, AR2599

This deliverable will define the requirements for data handling (quantitative and qualitative) throughout the entire “methodology chain” during an FOT: Data acquisition Data upload and quality assurance Database storage and data management Data analysis Valuable input for this work has come from international sources of FOT experience such as the University of Michigan Transportation Research Institute (UMTRI), Virginia Tech Transportation Institute, the Volpe centre, and others. Results presented here include data acquisition system (DAS) requirements. Different types of logging needs (for different purposes) will be considered, such as vehicle systems testing, real-time traffic information functions, and vehicle-toinfrastructure solutions. Experiences from state-of-the-art FOTs on DAS covered: sensors, video logging, vehicle dynamics, vehicle network logging, radar, driver behaviour, system performance, and qualitative data (interviews, questionnaires, and background information). Background information on involved concepts, as well as suggestions on how to arrive at FOT specific requirements, are provided. Aspects of OEM and systems provider co-operation and proprietary issues are handled, as well as issues with personal integrity and privacy. Further, technical information on data storage, database design, quality management procedures, and suggestions for data analysis tools is provided

FESTA Support Action Field opErational teSt supporT Action. D2.1 - A comprehensive framework of performance indicators and their interaction

The FESTA PI Matrix is a document containing Performance Indicators that can be used to assess safety, efficiency, environmental and acceptance aspects in a Field Operational Test (FOT). The list was compiled by a number of experts who used their own experience and the literature as basis. The list is meant to be used as a tool both during the planning phase and during the analysis phase of an FOT. It should also be of help for budget decisions, as it can aid the user in estimating sensor costs, for example, but also in estimating how intricate and time intensive certain analyses are. The list is meant to be used by people with background knowledge in the field, it does not substitute a solid education in traffic research. Even though the list is quite comprehensive, it is by no means exhaustive, which means that existing and established Performance Indicators might not be included, even though some effort has been made to cover all aspects that nowadays can be measured in a reasonable way in an FOT. The list can be extended and new Performance Indicators can be added

FESTA. D5 Common vision regarding nomadic systems FOTs. AR2605

On the basis of systems & functions identified in WP5 nomadic devices, the Nomadic Device team in FESTA has focused its efforts to find a generic method to answer these following questions: • What are the relevant nomadic devices services to be tested in an FOT? • What are the relevant use cases lists that correspond to mature systems? • How to identify & prepare the research & questions towards nomadic devices? • How to link hypotheses to indicators and concrete measurements during the FOT? In the annex, the reader will find a list answering partially these questions but most important the process was used to setup a methodology for the selection of the systems and functions and the identification of the research questions for future FOTs addressing Nomadic Devices. Therefore, this document provides a procedure on how to build a list of functions and hypotheses taking into account issues with a holistic point of view on safety, mobility, environment, business and implementation rather than providing an exhaustive list of functions or hypotheses which would need to be tested in an FOT. The main chapter of this document describes 5 steps to go from the definition of the relevant functions and systems to the hypotheses to be tested during an FOT. Eventually, the hypotheses will lead to a list of indicators and measures which will help the study design of your FOT. The five steps are: • Step 1: Selection and description of Functions • Step 2: Definition of use cases and situations • Step 3: Identification of the Research Questions • Step 4: Creation of Hypotheses • Step 5: Link Hypotheses with indicators for quantitative analyses Building the right list of functions and their hypotheses will be the basis of a good FOT. Understanding the political, societal and technical benefit of the FOT as a whole before it has even started is a key to its successful outcome. It is therefore strongly advised to put considerable effort and time on this preliminary work which might seems more paper-like before starting the exciting implementation of the devices and functions for large-scale testing